Image pickup apparatus, spectroscopic system, and spectroscopic method

ABSTRACT

An image pickup apparatus includes an encoder which is arranged on an optical path of light incident from an object and which has a plurality of regions with first light transmittance and a plurality of regions with second light transmittance lower than the first light transmittance, a dispersive element which is arranged on an optical path of at least one part of light after passage through the encoder and which spatially shifts the at least one part of the light in accordance with wavelength, and at least one image pickup device which is arranged to receive light after passage through the dispersive element and light without passage through the dispersive element and which acquires a first image, in which light components for respective wavelengths spatially shifted by the dispersive element are superimposed, and a second image based on the light without passage through the dispersive element.

BACKGROUND

1. Technical Field

The present disclosure relates to an image pickup apparatus foracquiring a spectral image, a spectroscopic system, and a spectroscopicmethod using the image pickup apparatus and/or the spectroscopic system.

2. Description of the Related Art

Utilization of spectral information for a large number (e.g., severaltens or more) of narrow wavelength bands allows grasp of detailedphysical properties of an observed object, which cannot be obtained froma conventional RGB image. A camera which acquires information formultiple wavelength bands is called a “hyperspectral camera”.Hyperspectral cameras are used in all fields, such as food inspection,living body inspection, drug development, and mineral componentanalysis.

As an example of utilization of information for a limited narrowwavelength band, International Publication No. 13/002350 discloses adevice which discriminates between a tumor site and a non-tumor site ina subject. The device detects, through irradiation with excitationlight, fluorescence at 635 nm emitted from protoporphyrin IX which isaccumulated in cancer cells and fluorescence at 675 nm emitted fromphoto-protoporphyrin, thereby distinguishing between a tumor site and anon-tumor site.

Japanese Unexamined Patent Application Publication No. 2007-108124discloses a method for judging the freshness of perishable food thatlowers over time by acquiring information on reflectance characteristicsof light with continuous multiple wavelengths.

Hyperspectral cameras capable of acquiring images and/or measuringreflectance for multiple wavelengths are broadly divided into the fourtypes:

-   (a) a line sensor type;-   (b) an electronic filter type;-   (c) a Fourier transform type; and-   (d) an interference filter type.

In a hyperspectral camera of the line sensor type in (a),one-dimensional information on an object is acquired using a memberhaving a linear slit. Light after passage through the slit is separatedby a dispersive element, such as a diffraction grating or a prism, inaccordance with wavelength. Separated light components of differentwavelengths are detected by an image pickup device (e.g., image sensor)having a plurality of two-dimensionally arrayed pixels. Under thissystem, only one-dimensional information on an object to be measured isobtained at one time. Two-dimensional spectral information is scanned byoperating the whole camera or the object to be measured perpendicularlyto a direction of the slit. The line sensor type has the advantage thathigh-resolution images for multiple wavelengths are obtained. JapaneseUnexamined Patent Application Publication No. 2011-89895 discloses anexample of the hyperspectral camera of the line sensor type.

Hyperspectral cameras of the electronic filter type in (b) fall into atype using a liquid crystal tunable filter (LCTF) and a type using anacousto-optic tunable filter (AOTF). A liquid crystal tunable filter isan element with multiple tiers, each having a linear polarizer, abirefringent filter, and a liquid crystal cell. The liquid crystaltunable filter can eliminate light of an unnecessary wavelength andextract only light of any specific wavelength only through voltagecontrol. An acousto-optic device is composed of an acousto-optic crystaland a piezoelectric element, which are bonded with each other. When anelectrical signal is applied to the acousto-optic crystal, ultrasonicwaves are generated, and thereby compressional standing waves are formedwithin the crystal. With a diffraction effect of the standing waves, theacousto-optic device can extract only light of any specific wavelength.Although a target wavelength is limited, the electronic filter type hasthe advantage of being able to acquire high-resolution moving imagedata.

A hyperspectral camera of the Fourier transform type in (c) uses theprinciple in a two-beam interferometer. A light beam from an object tobe measured is split into light beams by a beam splitter, and the lightbeams are reflected by a fixed mirror and a movable mirror,respectively, and are coupled again. A coupled light beam is thenobserved by a detector. Data indicating a change in the intensity ofinterference dependent on a light wavelength can be acquired by varyingthe position of the movable mirror over time. A Fourier transform isperformed on the obtained data to obtain spectral information. TheFourier transform type has the advantage of being able to simultaneouslyacquire pieces of information for multiple wavelengths.

A hyperspectral camera of the interference filter type in (d) is of atype using the principle in a Fabry-Perot interferometer. Theinterference filter type uses a configuration in which optical elementshaving two faces with high reflectance which are separate by apredetermined distance are arranged on a sensor. Distances between thetwo faces of the optical elements differ from region to region and aredetermined so as to meet an interference condition for light of adesired wavelength. The interference filter type has the advantage ofbeing able to simultaneously acquire pieces of information for multiplewavelengths as a moving image.

Besides the types, there is available a type using compressive sensing,as disclosed in, for example, U.S. Pat. No. 7,283,231. A devicedisclosed in U.S. Pat. No. 7,283,231 disperses light from an object tobe measured with a first dispersive element such as a prism, marks thedispersed light with a coded mask, and restores a light ray path with asecond dispersive element. With this configuration, an image which isobtained through coding and multiplexing on a wavelength axis isacquired by a sensor. A plurality of images for multiple wavelengths canbe reconstructed from the multiplexed image through use of compressionsensing.

Compressing sensing refers to a technique for restoring, from a smallnumber of pieces of data acquired as samples, pieces of data larger innumber. Assuming that (x,y) is two-dimensional coordinates of an objectto be measured and λ is a wavelength, data f desired to be obtained isthe three-dimensional data (x,y,λ). In contrast, image data g obtainedby a sensor is two-dimensional data which is compressed and multiplexedin a λ axis direction. The problem of obtaining the data f larger indata volume from the acquired image g smaller in data volume is aso-called ill-posed problem and is impossible to solve in this state.However, data of a natural image generally has redundancy, and skillfulutilization of the redundancy allows conversion of the ill-posed probleminto a well-posed problem. An example of a technique for reducing datavolume using the redundancy of an image is JPEG compression. JPEGcompression uses the process of converting image information into afrequency component and removing a non-essential portion of data (forexample, a component with low visibility). In compressive sensing, theabove-described technique is incorporated into arithmetic processing,and a desired data space is converted into a space represented withredundancy. With this conversion, unknowns are reduced, and a solutionis obtained. For the conversion, for example, a discrete cosinetransform (DCT), a wavelet transform, a Fourier transform, or totalvariation (TV) is used.

SUMMARY

One non-limiting and exemplary embodiment provides a new image pickuptechnique capable of simultaneously satisfying the three requirements:high resolution, multiple wavelengths, and moving image photographing(e.g., one-shot photographing).

In one general aspect, the techniques disclosed here feature an imagepickup apparatus including an encoder which is arranged on an opticalpath of light incident from an object and which has a plurality ofregions with first light transmittance and a plurality of regions withsecond light transmittance lower than the first light transmittance, adispersive element which is arranged on an optical path of at least onepart of light after passage through the encoder and which spatiallyshifts the at least one part of the light in accordance with wavelength,and at least one image pickup device which is arranged to receive lightafter passage through the dispersive element and light without passagethrough the dispersive element, the at least one image pickup deviceacquiring a first image, in which light components for respectivewavelengths spatially shifted by the dispersive element aresuperimposed, and a second image based on the light without the passagethrough the dispersive element.

It should be noted that comprehensive or specific embodiments may beimplemented as an apparatus, a device, a system, a method, or anyselective combination thereof.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an image pickup apparatus according toa first embodiment of the present disclosure;

FIG. 2A is front view of an encoder according to the first embodiment ofthe present disclosure when viewed from a subject side, which has anencoding pattern having a binary transmission/reflection distribution;

FIG. 2B is front view of an encoder according to the first embodiment ofthe present disclosure when viewed from a subject side, which has anencoding pattern having a gray-scale transmittance distribution;

FIG. 3 is a flowchart showing the overview of a spectroscopic methodaccording to the first embodiment of the present disclosure;

FIG. 4 is a schematic diagram showing an image pickup apparatusaccording to a second embodiment of the present disclosure;

FIG. 5 is a schematic diagram showing an image pickup apparatusaccording to a third embodiment of the present disclosure;

FIG. 6A is a view showing the relation between an optical array elementand an image pickup device according to the third embodiment of thepresent disclosure;

FIG. 6B is a view showing the positional relation between the opticalarray element and pixels on the image pickup device;

FIG. 7 is a schematic view showing dispersive elements according to afourth embodiment of the present disclosure;

FIG. 8A is a view showing the relation between an optical array elementand an image pickup device according to the fourth embodiment of thepresent disclosure;

FIG. 8B is a view showing the positional relation between the opticalarray element and pixels on the image pickup device;

FIG. 9 is a view showing an image generation result according to Exampleof the present disclosure;

FIG. 10 is a view showing an image generation result according toComparative Example of the present disclosure; and

FIG. 11 is a chart showing mean squared errors (MSEs) between respectivespectral images generated in Example and Comparative Example of thepresent disclosure and a correct answer image.

DETAILED DESCRIPTION Underlying Knowledge Forming Basis of the PresentDisclosure

Before describing embodiments of the present disclosure, findings madeby the present inventors will be described.

Studies by the present inventors have revealed that the above-describedconventional hyperspectral cameras suffer from the problems below. Thehyperspectral camera of the line sensor type in (a) needs to be operatedto obtain a two-dimensional image and therefore is unfit to photograph amoving image of an object to be measured. The hyperspectral camera ofthe Fourier transform type in (c) needs movement of a reflecting mirrorand therefore is unfit for moving image photographing. The hyperspectralcamera of the electronic filter type in (b) acquires an image for onewavelength at a time and therefore is incapable of simultaneouslyacquiring images for multiple wavelengths. The hyperspectral camera ofthe interference filter type in (d) has a tradeoff between a spatialresolution and the number of wavelength bands, and therefore images formultiple wavelengths cannot be acquired without scarifying the spatialresolution. As described above, there is no hyperspectral camera thatcan simultaneously satisfy the three requirements of high resolution,multiple wavelengths, and moving image photographing (e.g., one-shotphotographing), among existing hyperspectral cameras.

A configuration using compressive sensing is seemingly capable ofsimultaneously achieving high resolution, multiple wavelengths, andmoving image photographing. However, since an image is reconstructedfrom a small amount of data by inference, the spatial resolution of animage to be acquired is likely to be lower than that of an originalimage. In particular, the higher the compression ratio of acquired data,the more pronounced effect the compression ratio has on the spatialresolution.

The present inventors have found the above-described problems and thenstudied a configuration for solving the problems. The present inventorshave found that a reduction in resolution can be curbed by performingcomputation using not only an image, in which light components ofdifferent wavelengths are superimposed using a dispersive element, butalso an image acquired without dispersion, and thus completed thetechnique according to the present disclosure. According to embodimentsof the present disclosure, images for multiple wavelengths with aresolution higher than that of images for multiple wavelengths obtainedby a conventional method using compressive sensing can be acquired. Thisallows simultaneous satisfaction of the three requirements of highresolution, multiple wavelengths, and moving image photographing (e.g.,one-shot photographing). In an embodiment of the present disclosure, apiece of information for a wavelength direction can be compressed whiletwo-dimensional data, for an x direction and a y direction, is hold.This enables data volume to be reduced, thereby allowing long-time datato be acquired.

Overview of Embodiments

The present disclosure includes image pickup apparatuses, a system, anda method described in the items below.

[Item 1]

An image pickup apparatus including: an encoder which is arranged on anoptical path of light incident from an object and which has a pluralityof regions with first light transmittance and a plurality of regionswith second light transmittance lower than the first lighttransmittance; a dispersive element which is arranged on an optical pathof at least one part of light after passage through the encoder andwhich spatially shifts the at least one part of the light in accordancewith wavelength; and at least one image pickup device which is arrangedto receive light after passage through the dispersive element and lightwithout passage through the dispersive element, the at least one imagepickup device acquiring a first image, in which light components forrespective wavelengths spatially shifted by the dispersive element aresuperimposed, and a second image based on the light without passagethrough the dispersive element.

[Item 2]

The image pickup apparatus according to Item 1, further including: abeam splitter which is arranged on the optical path of the lightincident from the object and which transmits one part of the light andreflects another part of the light, in which the dispersive element isarranged on an optical path of one of the parts of the light obtainedthrough separation by the beam splitter, and the at least one imagepickup device includes a first image pickup device arranged to receivethe light after passage through the dispersive element and a secondimage pickup device arranged on an optical path of the other of theparts of the light obtained through the separation by the beam splitter,the first image pickup device acquires the first image, and the secondimage pickup device acquires the second image.

[Item 3]

The image pickup apparatus according to Item 2, in which the encoder isarranged between the beam splitter and the dispersive element.

[Item 4]

The image pickup apparatus according to Item 3, further including: afirst optical system which is arranged between the object and the beamsplitter and which focuses the light from the object onto a face of theencoder; and a second optical system which is arranged between theencoder and the dispersive element and which focuses the light afterpassage through the encoder onto an image pickup face of the first imagepickup device.

[Item 5]

The image pickup apparatus according to Item 2, in which the encoder isarranged between the object and the beam splitter, and the image pickupapparatus further includes an optical system which is arranged betweenthe encoder and the beam splitter and which focuses the light afterpassage through the encoder onto an image pickup face of the first imagepickup device and onto an image pickup face of the second image pickupdevice.

[Item 6]

The image pickup apparatus according to Item 1, in which the imagepickup device has a plurality of first light detection cells foracquiring the first image and a plurality of second light detectioncells for acquiring the second image, the dispersive element is arrangedon an optical path of one part of the light after passage through theencoder, and the image pickup apparatus further includes an opticalarray element which is arranged to face an image pickup face of theimage pickup device, the optical array element causing the light afterpassage through the dispersive element to enter the plurality of firstlight detection cells and causing the light without passage through thedispersive element to enter the plurality of second light detectioncells.

[Item 7]

The image pickup apparatus according to any one of Items 1 to 6, furtherincluding a signal processing circuit which generates a plurality ofimages for respective wavelength bands of the light after passagethrough the dispersive element on a basis of the first image, the secondimage, and a spatial distribution of light transmittance in the encoder.

[Item 8]

The image pickup apparatus according to Item 7, in which the signalprocessing circuit generates the plurality of images for the respectivewavelength bands by a statistical method.

[Item 9]

The image pickup apparatus according to Item 7 or 8, in which the numberof pieces of data in the plurality of images for the respectivewavelength bands of the light is larger than a sum of the number ofpieces of data in the first image and the number of pieces of data inthe second image.

[Item 10]

The image pickup apparatus according to any one of Items 7 to 9, inwhich the signal processing circuit generates, as the plurality ofimages for the respective wavelength bands, a vector f′ calculated usinga vector g having, as elements, signal values of a plurality of pixelsin the first image and signal values of a plurality of pixels in thesecond image and a matrix H determined by the spatial distribution oflight transmittance in the encoder and a spectral characteristic of thedispersive element, the vector f′ being calculated by:

$f^{\prime} = {{\underset{f}{argmin}{{g - {Hf}}}_{l_{2}}} + {{\tau\Phi}(f)}}$

where τΦ(f) is a regularization term and τ is a weighting factor.

[Item 11]

A spectroscopic system including: an image pickup apparatus whichincludes an encoder which is arranged on an optical path of lightincident from an object and which has a plurality of regions with firstlight transmittance and a plurality of regions with second lighttransmittance lower than the first light transmittance, a dispersiveelement which is arranged on an optical path of at least one part oflight after passage through the encoder and which spatially shifts theat least one part of the light in accordance with wavelength, and atleast one image pickup device which is arranged to receive light afterpassage through the dispersive element and light without passage throughthe dispersive element, the at least one image pickup device acquiring afirst image, in which light components for respective wavelengthsspatially shifted by the dispersive element are superimposed, and asecond image based on the light without passage through the dispersiveelement; and a signal processing apparatus which generates a pluralityof images for respective wavelength bands of light after passage throughthe dispersive element on a basis of the first image, the second image,and a spatial distribution of light transmittance in the encoder.

[Item 12]

A spectroscopic method including: spatially encoding intensity of lightincident from an object; spatially shifting at least one part of theencoded light in accordance with wavelength; acquiring a first image, inwhich light components for respective wavelengths after the spatialshifting are superimposed, and a second image based on light without thespatial shifting in accordance with wavelength; and generating aplurality of images for respective wavelength bands on a basis of thefirst image, the second image, and a pattern for the encoding.

More specific embodiments of the present disclosure will be describedbelow with reference to the drawings.

The embodiments described below are all comprehensive or specificexamples. Numerical values, shapes, materials, constituent elements,arrangement positions and connection forms of the constituent elements,steps, the order of the steps, and the like illustrated in theembodiments below are merely illustrative, and are not intended to limitthe present disclosure. Among the constituent elements in theembodiments below, those not described in an independent claimrepresenting a top-level concept will be described as optionalconstituent elements.

In the description below, signals representing an image (i.e., acollection of signals indicating respective pixel values of pixels) maybe simply referred to as an “image”. x, y, and z coordinates shown inthe drawings may be used in the description below.

First Embodiment

FIG. 1 is a schematic diagram showing an image pickup apparatus D1according to a first embodiment. The image pickup apparatus D1 accordingto the present embodiment includes imaging optical systems L1 and L2, abeam splitter B, an encoder C, a dispersive element P, and image pickupdevices S1 and S2. The image pickup device S1 acquires a first image G1and the image pickup device S2 acquires a first image G2. A signalprocessing circuit Pr is also shown in FIG. 1. The signal processingcircuit Pr processes image signals output from the image pickup devicesS1 and S2. The signal processing circuit Pr may be incorporated in theimage pickup apparatus D1 or may be provided outside the image pickupapparatus D1. In the latter case, a signal processing apparatus mayinclude the signal processing circuit Pr and the image pickup apparatusD1, which are electrically connected to each other by wire orwirelessly. The signal processing circuit Pr estimates, based on thefirst image G1 and the second image G2, a plurality of images F forrespective wavelength bands of light from an object O to be measured.The plurality of images F may also be referred to as “spectral images F”hereinafter.

The imaging optical systems L1 and L2 each include at least one imagepickup lens. Although the imaging optical systems L1 and L2 are eachshown as one lens in FIG. 1, the imaging optical systems L1 and L2 mayeach include a combination of a plurality of lenses. FIG. 1 showsoptical axis directions of the imaging optical systems L1 and L2 as az-axis.

The beam splitter B is arranged between the imaging optical system L1and the imaging optical system L2. The beam splitter B splits anincident light beam into two directions regardless of wavelength. Morespecifically, the beam splitter B transmits a part of a light beamincident from an object in a direction of the image pickup device S1 andreflects another part in a direction of the image pickup device S2.Examples of the beam splitter B include a cubic beam splitter made oftwo prisms and a beam splitter plate such as a half mirror, for example.

The encoder C is arranged at an imaging plane of the imaging opticalsystem L1. The encoder C is arranged between the beam splitter B and thedispersive element P. The encoder C is a mask which has a spatialdistribution of light transmittance. The encoder C has at least aplurality of regions with first light transmittance and a plurality ofregions with second light transmittance lower than the first lighttransmittance. The encoder C lets light incident through the beamsplitter B pass through while spatially modulates the intensity of thelight.

FIGS. 2A and 2B are views showing examples of a two-dimensionaldistribution of light transmittance of the encoder C. FIG. 2A shows alight transmittance distribution of the encoder C according to thepresent embodiment. In FIG. 2A, a black portion represents a regionwhich hardly transmits light (referred to as a “light-blocking region”)while a white portion represents a region which transmits light(referred to as a “translucent region”). In this example, the lighttransmittance of the white portion is almost 100%, and the lighttransmittance of the black portion is almost 0%. The encoder C isdivided into a plurality of rectangular regions, each of which is atranslucent region or a light-blocking region. Thus, in the exampleshown in FIG. 2A, the encoder C has a plurality of rectangular regionswith the first light transmittance of 100% and a plurality ofrectangular regions with the second light transmittance of almost 0%. Atwo-dimensional distribution of translucent regions and light-blockingregions in the encoder C can be, for example, a random distribution or aquasi-random distribution.

A random distribution and a quasi-random distribution are considered asfollows. Each rectangular region in the encoder C can be regarded as,for example, a vector element which has a value of 1 or 0 correspondingto light transmittance. In other words, a collection of rectangularregions arranged in a column can be regarded as a multi-dimensionalvector having values of 1 or 0. Thus, the encoder C includes a pluralityof multi-dimensional vectors in a row direction. In this case, a randomdistribution means that any two multi-dimensional vectors areindependent of (i.e., not parallel to) each other. A quasi-randomdistribution means that some multi-dimensional vectors include oneswhich are not independent.

A random distribution and a quasi-random distribution can also bedefined using an autocorrelation function defined by the Formula (1):

$\begin{matrix}{{y\left( {i,j} \right)} = {\sum\limits_{m = 1}^{M}\; {\sum\limits_{n = 1}^{N}\; {{x\left( {m,n} \right)} \cdot {x\left( {{m + i},{n + j}} \right)}}}}} & (1)\end{matrix}$

In Formula (1), x(m,n) represents the light transmittance of arectangular region which is the m-th in a longitudinal direction and then-th in a lateral direction in the encoder C composed of an array of Mwide by N high rectangular regions, that is, a total of M×N rectangularregions. Additionally, i=−(M−1), . . . , −1, 0, 1, . . . , (M−1), andj=−(N−1), . . . , −1, 0, 1, . . . , (N−1). Note that if m<1, n<1, m>M,or n>N, x(m,n)=0 is obtained. In this case, a random distribution meansthat the autocorrelation function y(i,j) defined by Formula (1) has amaximum value if i=0 and j=0 and does not have a maximum value otherwise(if i≠0 and/or j≠0). More specifically, a random distribution means thatthe autocorrelation function y(i,j) decreases monotonically from i=0toward i=M−1 and i=−(M−1) and decreases monotonically from j=0 towardj=N−1 and j=−(N−1). A quasi-random distribution means that theautocorrelation function y(i,j) has a maximum value at M/10 or lesspoints in an i direction and a maximum value at N/10 or less points in aj direction, besides a point for y(0,0).

The light passing through the encoder C is dispersed by the dispersiveelement P to form an optical image, in which light components ofdifferent wavelengths are imaged in mutually shifted positions.Therefore encoding by the encoder C plays the role of marking fordistinguishing among the light components of different wavelengths inthe image. The distribution of transmittance may be arbitrarily set aslong as the above-described marking is possible. Although the ratiobetween the number of black portions and that of white portions is 1:1in the example shown in FIG. 2A, the present disclosure is not limitedto this. For example, the distribution may be deflected to one side,like a case where the ratio between the number of white portions andthat of black portions is 1:9. As shown in FIG. 2B, the mask may be amask having a gray-scale distribution of transmittance. In this case,the encoder C has a plurality of rectangular regions with third lighttransmittance different from the first and second light transmittance,for example. Information on the transmittance distribution of theencoder C may be acquired in advance from design data or by actualmeasurement calibration.

The dispersive element P is an element which disperses an incident lightbeam in accordance with wavelength. The dispersive element P can becomposed of, for example, a prism or a diffractive optical element.Light encoded by the encoder C passes through the imaging optical systemL2 and then enters the dispersive element P. The light dispersed by thedispersive element P forms the optical image, which contains lightcomponents of different wavelengths, on an image pickup face of theimage pickup device S1. In FIG. 1, the dispersive element P causesimaging positions of wavelength components included in the optical imageto be mutually shifted in a y direction (i.e., a longitudinal directionof the optical image) in accordance with wavelength. If the dispersiveelement P is a prism, the shift amount is determined by the refractiveindex of the dispersive element P, an Abbe number of the dispersiveelement P, a surface tilt angle of the dispersive element P, and thedistance between the dispersive element P and the image pickup deviceS1. If the dispersive element P is a diffractive optical element, theshift amount is determined by the refractive index of the dispersiveelement P, the Abbe number of the dispersive element P, the distancebetween the dispersive element P and the image pickup device S1, and adiffractive grating pitch. Here, the term “shift amount” means a degreeof relative positional shift among wavelength components in an opticalimage. The term “shift direction” means a direction of relativepositional shift among wavelength components in an optical image. Thedispersive element P may cause imaging positions of wavelengthcomponents in the optical image to be mutually shifted in an x direction(i.e., a lateral direction of the optical image) or in anotherdirection. Alternatively, the shift amount or the shift direction may bechanged in accordance with the object O to be measured. In the case of,for example, the object O to be measured that has a high-frequencytexture (for example, lateral stripes) in the y direction, thedispersive element P may be configured such that the shift amount in they direction is larger or the shift direction is not the y direction butthe x direction.

The shift amounts for wavelengths take not discrete values for eachwavelength band but continuous values according to wavelength. The shiftamount may be calculated in advance by computation from designspecifications or by actual measurement calibration. Note that thespectral images F are reconstructed for respective wavelength bands ofpredetermined width by the signal processing circuit Pr (to be describedlater). To be exact, wavelength components of the optical image are,even if belonging to a wavelength band of a spectral image F, imaged onthe image pickup device S1 in mutually shifted positions. To improve theaccuracy of reconstruction of the spectral images F, it is desirable tocorrect the positional shift among the light components which belong toa common wavelength band. The correction of the shift may be performedby computer arithmetic, or desirably by actual measurement calibrationin view of effects of an optical system aberration and/or a mountingerror. For example, calibration can be performed by installing a whiteplate as a subject at the position of the object O to be measured andforming an image of the encoder C on the image pickup device S1 througha band-pass filter for a desired wavelength band. Pieces of data for alldesired wavelength bands may be acquired by using different band-passfilters for respective wavelength bands. Alternately, measurement may beperformed for some selected bands, and pieces of data for the otherbands between measured pieces of data may be interpolated. This methodallows calculation of the shift amounts for wavelengths and also allowsacquisition of pieces of transmittance information of the encoder C forthe respective wavelength bands. Elements of a matrix H in Formulas (2)(to be described later) are determined on the basis of pieces of datacalculated by calibration, for example.

The image pickup devices S1 and S2 are each a monochrome image pickupdevice having a plurality of two-dimensionally arrayed light detectioncells (referred to as “pixels” in the present specification). The imagepickup devices S1 and S2 can be, for example, charge-coupled device(CCD) or complementary metal oxide semiconductor (CMOS) sensors. Eachlight detection cell can be composed of, for example, a photodiode. Theimage pickup devices S1 and S2 may not be monochrome image pickupdevices. For example, the image pickup devices S1 and S2 may be colorimage pickup devices having R, G, and B filters, R, G, B, and IRfilters, or R, G, B, and W filters. The use of a color image pickupdevice allows an increase in the amount of wavelength-relatedinformation and thereby improvement in the accuracy of reconstruction ofthe spectral images F. Note that, if a color image pickup device isused, the amount of information in spatial directions (e.g., the x and ydirections) decreases, because there is a trade-off between the amountof wavelength-related information and resolution. A measuring wavelengthrange of the image pickup devices S1 and S2 may be arbitrarily set. Thewavelength range is not limited to a visible wavelength range and may bean ultraviolet, near-infrared, mid-infrared, or far-infrared wavelengthrange.

The signal processing circuit Pr is a circuit which processes imagesignals input from the image pickup devices S1 and S2. The signalprocessing circuit Pr can be implemented by, for example, a programmablelogic device (PLD), such as a digital signal processor (DSP) or afield-programmable gate array (FPGA), or a combination of a centralprocessing unit (CPU) and an image processing arithmetic processor (GPU)and a computer program. Such a computer program is stored in a recordingmedium, such as a memory, and arithmetic processing (to be describedlater) can be executed by a CPU executing the program. As describedearlier, the signal processing circuit Pr may be provided outside theimage pickup apparatus D1. In this configuration, the signal processingcircuit Pr is contained in a signal processing apparatus, such as apersonal computer (PC) electrically connected to the image pickupapparatus D1 or a cloud server on the Internet. In the presentspecification, a system including the above-described signal processingapparatus and an image pickup apparatus will be referred to as a“spectroscopic system”.

The operation of the image pickup apparatus D1 according to the presentembodiment will be described below.

FIG. 3 is a flowchart showing the overview of a spectroscopic methodaccording to the present embodiment. In step S101, light from the objectO to be measured is encoded using the encoder C. The encoding isimplemented through spatial modulation of the intensity of the lightwith the spatial distribution of light transmittance of the encoder C.In step S102, the encoded light is dispersed in accordance withwavelength by the dispersive element P and the dispersed light is imagedon the image pickup face. In step S103, a first image G1, in whichcomponents of the dispersed light are superimposed in mutually shiftedpositions, is acquired by the image pickup device S1. A second image G2,based on light which is not spatially dispersed by the dispersiveelement P in accordance with wavelength, is acquired by the image pickupdevice S2. In step S104, the spectral images F are reconstructed on thebasis of the first image G1, the second image G2, and the lighttransmittance distribution of the encoder C.

A process of acquiring the first image G1 and the second image G2 by theimage pickup apparatus D1 according to the present embodiment will bedescribed.

A process of acquiring the first image G1 will be described first. Alight beam from the object O is focused on the encoder C by the imagingoptical system L1. An image of the light beam is encoded by the encoderC. In other words, the intensity of the light passing through theencoder C is modulated in accordance with the spatial distribution oftransmittance of the encoder C. The encoded light beam is focused againby the imaging optical system L2 to form an optical image on the imagepickup face of the image pickup device S1. At that time, the light beamis, before reaching the image pickup face, dispersed by the dispersiveelement P in accordance with wavelength. Thus, the light beam forms themultiple image, components of which overlap with one another in mutuallyshifted positions in accordance with wavelength, on the image pickupface of the image pickup device S1. The image pickup device S1 acquiresthe first image G1 by converting information of the multiple image intoa plurality of electrical signals (i.e., pixel signals) with theplurality of light detection cells thereof. In FIG. 1, shift amounts inthe y direction in the first image G1 are shown to be larger than actualsizes for the sake of clarity. A shift amount between images for twoadjacent wavelength bands can practically be, for example, as fine asone pixel to about several tens of pixels. In FIG. 1, the first image G1includes a plurality of black dots, which schematically representlow-brightness portions generated by the encoding. Note that the numberand the layout of black dots shown in FIG. 1 do not reflect an actualnumber and an actual layout. In practice, low-brightness portions largerin number than the black dots shown in FIG. 1 may appear.

A process of acquiring the second image G2 will be described. The lightbeam from the object O to be measured is split by the beam splitter Bafter passing through the imaging optical system L1. The split lightbeam forms an optical image on an image pickup face of the image pickupdevice S2. This optical image is based on a light beam without passagethrough the dispersive element P, that is, without spatially dispersedby the dispersive element P. The image pickup device S2 acquires thesecond image G2 by converting information of the optical image into aplurality of pixel signals with the plurality of light detection cellsthereof. The second image G2 is a monochrome image having informationfor all wavelength bands of the object to be measured. The second imageG2 is acquired almost at the same time as acquisition of the first imageG1.

In the example shown in FIG. 1, the first image G1 is formed from alight beam transmitted through the beam splitter B, and the second imageG2 is formed from a light beam reflected by the beam splitter B.However, light beams transmitted and reflected by the beam splitter Band the first and second images G1 and G2 may be arbitrarily combined.The first image G1 may be formed from a light beam reflected by the beamsplitter B, and the second image G2 may be formed from a transmittedlight beam.

The image pickup apparatus D1 may further include a band-pass filterwhich transmits only components with some wavelength bands of anincident light beam. With this configuration, a measurement wavelengthband can be limited. The limitation of the measurement wavelength bandallows acquisition of the spectral images F for a limited desiredwavelength with high separation accuracy.

A method for reconstructing the spectral images F for multiplewavelengths from the first image G1 and the second image G2 will bedescribed. The term multiple wavelengths refers to, for example,wavelength bands larger in number than wavelength bands for three colors(R, G, B) acquired by a general color camera. The number of wavelengthbands (that may be referred to as the “number of spectral bands”) canbe, for example, a number of four to about 100. The number of spectralbands may exceed 100 depending on the intended use.

Assuming that f is data of the spectral images F, and w is the number ofspectral bands, the data f is data into which pieces f₁, f₂, . . . ,f_(w) of image data for respective bands are integrated. Assuming that nis the number of pixels in an x direction of each piece of image data,and m is the number of pixels in a y direction, the pieces f₁, f₂, . . ., f_(w) of image data are a collection of pieces of two-dimensional datawith n×m pixels. Thus, the data f is a piece of three-dimensional datawith n×m×w elements. If components in a multiple image are shifted inthe y direction per one pixel for each spectral band, the number ofelements of data g₁ of the first image G1 to be acquired is n×(m+w−1).In contrast, the number of elements of data g₂ of the second image G2 isn×m. The data g₁ and the data g₂ in the present embodiment can berepresented by Formulas (2) below:

$\begin{matrix}{g_{1} = {{{H_{1}\begin{bmatrix}f_{1} \\f_{2} \\\vdots \\f_{w}\end{bmatrix}} g_{2}} = {H_{2}\begin{bmatrix}f_{1} \\f_{2} \\\vdots \\f_{w}\end{bmatrix}}}} & (2)\end{matrix}$

Since the pieces f₁, f₂, . . . , f_(w) of data are each a piece of datawith n×m elements, each vector on the right side is technically aone-dimensional vector with n×m×w rows and one column. Matrices H₁ andH₂ are system matrices indicating a conversion process from the data finto the data g₁ and a conversion process from the data f into the datag₂, respectively. The matrix H₁ indicates a conversion that encodes thedata f, that is, performs intensity modulation for each pixel, shiftsthe pixel values of the components f₁, f₂, . . . , f_(w) in the ydirection such that the pixel value of each component is shifted by onepixel with respect to the pixel value of a previous one, and adds up thepixel values. The matrix H₂ is a conversion that adds up the pixelvalues of the components f₁, f₂, . . . , f_(w) of the data f if atransmission/reflection split ratio at the beam splitter B is 1:1.Assuming that H is a system matrix into which the matrices H₁ and H₂ areintegrated, and g is data into which the data g₁ and the data g₂ areintegrated, the data g is represented by Formula (3) below:

g=Hf   (3)

Technically, the data f is expressed as a one-dimensional vector withn×m×w rows and one column, and the data g is expressed as aone-dimensional vector with n×(2m+w−1) rows and one column. Thus, thematrix H is a matrix with n×(2m+w−1) rows and n×m×w columns.

Since it is assumed in the present embodiment that images for respectivewavelength bands are shifted in increments of one pixel, the number ofelements of the data g₁ is n×(m+w−1). However, the images may not beshifted in increments of one pixel. A pixel increment for shifting maybe two or more pixels. The pixel increment for shifting depends on howspectral bands and the number of spectral bands for the spectral imagesF to be reconstructed are designed. The number of elements of the datag₁ changes in accordance with the pixel increment for shifting. Theshift direction is not limited to the y direction and may be the xdirection. In a generalized case, if an image is shifted in incrementsof ky pixels in the y direction and in increments of kx pixels in the xdirection, where ky and kx are arbitrary natural numbers, the number ofelements of the data g₁ is {n+kx·(w−1)}×{m+ky·(w−1)}.

The transmission/reflection split ratio at the beam splitter B may notbe 1:1 and may be any other value. If the split ratio is not 1:1, acoefficient for the matrix H₁ or the matrix H₂ of the system Formulas(2) may be corrected in accordance with the split ratio.

If the vector g and the matrix H are given, it seems that the data f canbe calculated by solving an inverse problem of Formula (3). However,since the number of elements of n×m×w of the desired data f is largerthan the number of elements of n×(2m+w−1) of the acquired data g, theproblem is an ill-posed problem and cannot be directly solved. For thisreason, the signal processing circuit Pr according to the presentembodiment uses image redundancy included in the data f and finds asolution using compressive sensing. More specifically, the desired dataf is estimated by solving Formula (4) below:

$\begin{matrix}{f^{\prime} = {\underset{f}{argmin}\left\{ {{{g - {Hf}}}_{l_{2}} + {{\tau\Phi}(f)}} \right\}}} & (4)\end{matrix}$

Reference character f′ represents estimated data of the data f. Thefirst term in the braces in Formula (4) represents the amount ofdeviation of an estimation result Hf from the acquired data g, that is,a so-called residual term. The residual term can be expressed as the sumof squares, an absolute value, or the square root of sum of squares. Thesecond term in the braces is a regularization term, or a stabilizationterm, (to be described later). Formula (4) means obtaining the data fthat minimizes the sum of the first and second terms. The signalprocessing circuit Pr can converge on a solution by recursive anditerative operations and thereby calculate the final solution f′.

The first term in the braces in Formula (4) means an operation ofcalculating the sum of squares of the difference between the acquireddata g and the estimation result Hf that is a result of subjecting thedata f to system conversion with the matrix H in the estimation process.Φ(f) in the second term denotes a constraint condition in regularizationof the data f and a function reflecting sparse information of estimateddata. The function Φ(f) has the effect of smoothing or stabilizingestimated data. The regularization term can be expressed as, forexample, the discrete cosine transform (DCT), the wavelet transform, theFourier transform, or the total variation (TV) of the data f. Forexample, if total variation is used as the regularization term, stableestimated data with reduced effect of noise in the observed data g canbe acquired. The sparseness of the object O to be measured in a space ofeach regularization term depends on the texture of the object O to bemeasured. A regularization term may be selected such that the texture ofthe object O to be measured is more sparse in a space of theregularization term. Alternatively, the operation may include aplurality of regularization terms. τ denotes a weighting factor. Thelarger a value of the weighting factor, the larger the amount ofredundant data cut (i.e., the higher a compression ratio). The smallerthe value of the weighting factor, the weaker convergence on a solution.The weighting factor τ is set to an appropriate value at which the dataf converges to some degree, and excessive compression is inhibited.

Note that although an operation example using compressive sensingindicated by Formula (4) has been illustrated here, a solution may befound using any other method. For example, a different statisticalmethod, such as maximum likelihood estimation or Bayes estimation, canbe used instead. The number of spectral images F may be any number, andwavelength bands may be arbitrarily set.

As described above, in the present embodiment, the image data f isobtained using not only an image including a plurality of wavelengthcomponents which are imaged in mutually shifted positions by thedispersive element P but also an image based on a light beam withoutpassage through the dispersive element P. As will be described inExample (to be described later), this configuration allows a reductionin deterioration of resolution due to compressive sensing. Thus, thethree requirements of high resolution, multiple wavelengths, and movingimage photographing (e.g., one-shot photographing) can be simultaneouslysatisfied. Since it suffices to hold two-dimensional data at the time ofimage pickup, the present embodiment is effective in long-time dataacquisition. Note that an image pickup device and a signal processingcircuit according to the present embodiment may be configured to acquireonly a still image.

Second Embodiment

A second embodiment is different from the first embodiment in that aspectral image is reconstructed using a blurred state of an encodingpattern in an image on an image face. A detailed description of the samematters as those in the first embodiment will be omitted below.

FIG. 4 is a schematic diagram showing an image pickup apparatus D2according to the present embodiment. In the image pickup apparatus D2,an encoder C is arranged between an object O to be measured and animaging optical system L, unlike the image pickup apparatus D1. Theimaging optical system L focuses a light beam after passage through theencoder C onto image pickup faces of image pickup devices S1 and S2.Since the encoder C may not be arranged at an imaging plane of animaging optical system in the present embodiment, an optical system(e.g., relay optical system) other than the imaging optical system L isunnecessary. This allows a reduction in the overall size of an opticalsystem.

The image pickup devices S1 and S2 acquire an encoding pattern of theencoder C in a blurred state. Thus, blur information is held in advanceand is reflected in the system matrix H in Formula (3). The blurinformation is represented by a point spread function (PSF) here. ThePSF is a function defining the degree of spread of a point image tosurrounding pixels. For example, if a point image corresponding to onepixel on an image spreads across a region with k×k pixels around thepixel due to blurring, the PSF can be defined as a group of coefficients(i.e., a matrix) which indicates effects on the brightness of the pixelswithin the region. Spectral images F can be reconstructed by reflectingeffects of the PSF on the encoding pattern of the encoder C in thesystem matrix H.

The encoder C may be arranged at any position, as far as not causing theencoding pattern of the encoder C to be severely blurred to disappear.For example, the encoder C may be arranged in the vicinity of a lensclosest to the object O to be measured in the imaging optical system Lor in the vicinity of the object O to be measured, and may be arrangedaway from a diaphragm. With this configuration, when the optical systemL has a wide angle of view and a short focal length, there is littleoverlap between light beams with respective angles of view, andtherefore the encoding pattern is likely to remain on the image pickupdevice S1 with little blur. Alternately, the encoder C may be arrangedcloser to the image pickup device S1. With this configuration, theencoding pattern is also likely to remain. Note that, in this case aswell, the encoder C needs to be arranged closer to the side of theobject O to be measured than a dispersive element P.

Third Embodiment

A third embodiment is different from the first embodiment in that afirst image G1 and a second image G2 are acquired using a configurationcalled pupil division. In the present embodiment, a detailed descriptionof the same matters as those in the first embodiment will be omitted.

FIG. 5 is a schematic diagram showing an image pickup apparatus D3according to the present embodiment. The image pickup apparatus D3 hasone image pickup device S and does not have a beam splitter B. In thepresent embodiment, the dispersive element P is arranged on an opticalpath of a part of a light beam after passage through an encoder C. Forexample, a dispersive element P is arranged at an area of a diaphragmface of an imaging optical system L2. An optical array element A isarranged directly on an image pickup face of the image pickup device S.a dispersive element P is arranged at a portion of a diaphragm face ofan imaging optical system L2.

FIG. 6A is a cross-sectional view showing the detailed configuration ofthe optical array element A. The optical array element A can be composedof, for example, a lenticular lens. The optical array element Aaccording to the present embodiment has a structure in which a pluralityof optical components M2 extending in an x direction are arrayed in a ydirection. The plurality of optical components M2 may be a plurality ofcylindrical lenses, for example. The present disclosure, however, is notlimited to this configuration. For example, the optical array element Amay have a configuration in which a plurality of optical components M2extending in the y direction are arrayed in the x direction.

The optical array element A is arranged in the vicinity of an imagingface of the imaging optical system L2. As shown in FIG. 6A, the opticalarray element A is spaced from an image pickup face Ni.

FIG. 6B is a plan view schematically showing some of a plurality oflight detection cells (e.g., pixels) Ph which are two-dimensionallyarrayed on the image pickup face of the image pickup device S. Theplurality of light detection cells are classified into a plurality offirst light detection cells (e.g., first pixels) Ph1 and a plurality ofsecond light detection cells (e.g., second pixels) Ph2. The first pixelPh1 is a pixel for acquiring a first image based on light after passagethrough the dispersive element P. The second pixel Ph2 is a pixel foracquiring a second image based on light without passage through thedispersive element P.

Only one of light beams R1 and R2 after passage through two opticalregions of the diaphragm face of the imaging optical system L2 passesthrough the dispersive element P. In other word, the diaphragm face ofthe imaging optical system L2 has an area where the dispersive element Pis located and an area where no dispersive element is located. Forexample, only the light beam R1 passes through the dispersive element P.After that, the light beams R1 and R2 enter the optical array element A.The optical array element A causes the light beam R1 after passagethrough the dispersive element P to enter a plurality of first pixelsPh1 in the image pickup device S and causes the light beam R2 withoutpassage through the dispersive element P to enter a plurality of secondpixels Ph2 in the image pickup device S.

FIG. 6B shows the positional relation between the optical array elementA and pixels on the image pickup device S. Each bold line indicates theborder between each adjacent two of the plurality of optical componentsM2 in the optical array element A. The optical array element A isarranged such that a surface having the plurality of optical componentsM2 formed thereon faces toward the image pickup face Ni. The pixels Phare arranged in a matrix at the image pickup face Ni.

The pixels Ph1 are arranged in a row in a lateral direction (i.e., rowdirection). The pixel Ph1 is arranged in every other row in alongitudinal direction (i.e., column direction). The pixels Ph2 arearranged in a row in the lateral direction. The pixel Ph2 is arranged inevery other row in the longitudinal direction. Rows with the pixels Ph1and rows with the pixels Ph2 are alternately arranged in thelongitudinal direction.

If the optical array element A has a structure in which the plurality ofoptical components M2, each extending in the x direction, are arrayed inthe y direction, the pixels Ph1 and the pixels Ph2 are arranged side byside in the x direction and alternately in the y direction. In thiscase, in the diaphragm face of the imaging optical system L2, theoptical region where the dispersive element P is arranged and theoptical region without a dispersive element are arranged in the ydirection. If the optical array element A has a configuration in whichthe plurality of optical components M2, each extending in the ydirection, are arrayed in the x direction, the pixels Ph1 and the pixelsPh2 are arranged side by side in the y direction and alternately in thex direction. In this case, in the diaphragm face of the imaging opticalsystem L2, the optical face region where the dispersive element P isarranged and the optical face region without a dispersive element arearranged in the x direction. Each of the pixel Ph1 and the pixel Ph2 maynot be arranged in every other row in the longitudinal direction and maybe arranged in every n-th (n≧1) row.

The optical array element A is arranged such that one of the opticalcomponents M2 corresponds to pixels in 2n rows, which are composed of ann-th row of pixels Ph1 and an n-th row of pixels Ph2, on the imagepickup face Ni. On the image pickup face Ni, microlenses M1 are providedso as to cover the faces of the pixels Ph1 and Ph2.

The above-described configuration allows the light beam R1 after passagethrough the dispersive element P to enter a plurality of first pixelsPh1 and allows the light beam R2 without passage through the dispersiveelement P to enter a plurality of second pixels Ph2. A signal processingcircuit Pr acquires, from the image pickup device S, a first image G1which is composed of signals output from the plurality of first pixelsPh1 and a second image G2 which is composed of signals output from theplurality of second pixels Ph2. Spectral images F can be estimated onthe basis of the first image G1 and the second image G2 in the samemanner as in the first embodiment.

In the present embodiment, the dispersive element P is provided in thevicinity of one of halves, into which the imaging optical system L2 isdivided, and two images are acquired. That is, a pupil division numberis two. The pupil division number, however, may be three or more.

Note that the number of pixels in the y direction is divided by thepupil division number and thus is reduced in the configuration accordingto the present embodiment, unlike the first embodiment. For example, ifthe pupil division number is two, the number of pixels is halved.

Fourth Embodiment

A fourth embodiment is different from the third embodiment in that twoor more dispersive elements P are arranged. In the present embodiment, adetailed description of the same matters as those in the thirdembodiment will be omitted.

FIG. 7 is a schematic view showing dispersive elements P1 and P2according to the present embodiment. FIG. 7 shows four parts of a regionin the vicinity of a diaphragm. This region corresponds to a region at asurface on the image pickup device S side of an imaging optical systemL2 shown in FIG. 5, for example. The dispersive element P1 is arrangedin at least one of the four parts, and the dispersive element P2 isarranged in at least another one. The dispersive element P1 disperses anincident light beam in a y direction in accordance with wavelength, andthe dispersive element P2 disperses an incident light beam in an xdirection.

FIG. 8A is a cross-sectional view showing the detailed configuration ofan optical array element A according to the present embodiment. In thepresent embodiment, the optical component M2 of the optical arrayelement A is a microlens. The use of a microlens as the opticalcomponent M2 allows a light beam incident on each microlens to beseparated in four directions.

FIG. 8B is a view showing the pixel configuration of an image pickupdevice S according to the present embodiment. The image pickup device Shas a plurality of pixels Ph1, on which a light beam after passagethrough the dispersive element P1 is incident, a plurality of pixelsPh3, on which a light beam after passage through the dispersive elementP2 is incident, and a plurality of pixels Ph2, on which a light beamwithout passage through any dispersive element is incident. The imagepickup device S acquires three different images through those pixels.

In the present embodiment, use of pieces of spectral data for both the xdirection and the y direction for reconstruction allows improvement inreconstruction accuracy. The configuration is particularly effective inthe case when the pattern of an object to be measured has no change inone direction (for example, a stripe pattern).

Shift amounts in accordance with the dispersive elements P1 and P2 maybe set different from each other. This allows acquisition of two typesof image data, image data with a larger number of spectral bands andimage data with a smaller number of spectral bands. For example, it ispossible to acquire a hyperspectral image with several tens of bands ormore from a light beam after passage through the dispersive element P1and a multi-spectral image with several bands from a light beam afterpassage through the dispersive element P2. Note that, in this case,shift directions by the dispersive elements P1 and P2 may not bedifferent. The shift directions of both of the dispersive elements P1and P2 may be the x direction or the y direction.

The present embodiment has illustrated an example in which a pupildivision number is four. The present disclosure, however, is not limitedto this. For example, the pupil division number may be two, three, orfive or more. In the present embodiment as well as the third embodiment,the number of pixels decreases with an increase in the pupil divisionnumber.

Experimental Results EXAMPLE

An example of the present disclosure will be described.

FIG. 9 is a view showing one example of a result of reconstructingspectral images F using a spectroscopic method according to the presentdisclosure. An element having a binary pattern with a plurality oftranslucent regions and a plurality of light-blocking regions randomlyarrayed, as shown in FIG. 2A, was used as an encoder C. The lighttransmittance of each translucent region was approximately 100%, and thelight transmittance of each light-blocking region was approximately 0%.The number of spectral bands was 20. A dispersive element P was designedso that components in a multiple image were shifted in a y direction perone pixel for each spectral band.

A first image G1 was a 500×292 pixel image which was formed from one oflight beams, into which a light beam is split by a beam splitter B. Thefirst image G1 was an image based on a light beam encoded and dispersedin accordance with wavelength. Images for the 20 wavelength bands wereshifted in an image longitudinal direction per one pixel for eachwavelength band. With the 20 bands, there was a spatial difference of 19pixels in the longitudinal direction between the image for a shortestwavelength band and the image for a longest wavelength band.

A second image G2 was a 500×292 pixel monochrome image which was formedfrom the other of the light beams split by the beam splitter B. Thesecond image G2 was an image based on a light beam without encoding ordispersing in accordance with wavelength and included pieces ofinformation for all the wavelength bands.

In Example, the spectral images F for 20 wavelength bands were obtainedby finding a solution by an estimation algorithm in Formula (4) usingthe first image G1 and the second image G2. At the time, total variation(TV) was used as a regularization term.

COMPARATIVE EXAMPLE

As a comparative example, spectral images F were reconstructed usingonly a first image G1 without use of a monochrome image as the secondimage G2 in Example.

FIG. 10 is a view showing a result of the comparative example. It can beseen that, although separation among spectral bands is satisfactory, aresolution is lower than the result in FIG. 9.

FIG. 11 shows respective mean squared errors (MSEs) with respect to acorrect answer image for Example and Comparative Example. Each MSE isrepresented by Formula (5) and indicates a mean square error per pixel.A smaller value means greater closeness to a correct answer image.

$\begin{matrix}{{MSE} = {\frac{1}{n \cdot m}{\sum\limits_{i = 1}^{n}\; {\sum\limits_{j = 1}^{m}\; \left( {I_{i,j}^{\prime} - I_{i,j}} \right)^{2}}}}} & (5)\end{matrix}$

In Formula (5), n and m represent the numbers of longitudinal pixels andlateral pixels, respectively, of an image, I′_(i,j) represents a pixelvalue in an i-th row and a j-th column of a reconstructed image (i.e.,spectral image), and I_(i,j) represents a pixel value in an i-th row anda j-th column of a correct answer image. Note that images used inExample and the Comparative Example are 8-bit images, and the maximumvalue of a pixel value is 255.

The abscissa in FIG. 11 represents the image number of eachreconstructed spectral image F, and the ordinate represents the value ofan MSE. It can be confirmed from FIG. 11 that, if reconstruction isperformed by the method according to Example, the MSE of any spectralimage F has a small value and that the spectral image F is close to thecorrect answer image. The value of the MSE is about 10 to 20, and thespectral image F is found to almost coincide with the correct answerimage. In Comparative Example using a conventional method, it can beseen that the values of MSEs are uniformly high and that the spectralimages are more severely degraded than the correct answer image. An MSEaccording to Example is 12 times smaller than an MSE according toComparative Example at a minimum and is 34 times smaller at a maximum,which shows the efficacy of the method according to the presentdisclosure.

An image pickup apparatus according to the present disclosure iseffective for a camera or a measurement instrument which acquirestwo-dimensional images with multiple wavelengths. The image pickupapparatus can be applied to sensing for a living body, medical purposes,and cosmetic purposes, a system for inspecting foreign substances andresidual pesticides in food, a remote sensing system, an in-car sensingsystem, and the like.

While the present disclosure has been described with respect toexemplary embodiments thereof, it will be apparent to those skilled inthe art that the disclosure may be modified in numerous ways and mayassume many embodiments other than those specifically described above.Accordingly, it is intended by the appended claims to cover allmodifications of the disclosure that fall within the true spirit andscope of the disclosure.

What is claimed is:
 1. An image pickup apparatus comprising: an encoderlocated on an optical path of light from an object, the encoderincluding a plurality of first regions with first light transmittanceand a plurality of second regions with second light transmittance lowerthan the first light transmittance; a dispersive element located fartheraway from the object than the encoder on the optical path to cause lightto be spatially dispersed into a plurality of light components ofdifferent wavelengths; and at least one image pickup device located toreceive the plurality of light components of different wavelengths toacquire a first image, in which the plurality of light components ofdifferent wavelengths are superimposed in mutually shifted positions,and receive light from the object without passing through the dispersiveelement to acquire a second image.
 2. The image pickup apparatusaccording to claim 1, further comprising: a beam splitter locatedbetween the object and the dispersive element on the optical path tosplit light incident thereto into a plurality of split light, whereinthe dispersive element causes one of the plurality of split light to bespatially dispersed into the plurality of light components of differentwavelengths, and the at least one image pickup device includes a firstimage pickup device located to receive the plurality of light componentsof different wavelengths to acquire the first image, and a second imagepickup device located to receive another of the plurality of split lightwithout passing through the dispersive element to acquire the secondimage.
 3. The image pickup apparatus according to claim 2, wherein theencoder is located between the beam splitter and the dispersive element.4. The image pickup apparatus according to claim 3, further comprising:a first optical system located between the object and the beam splitteron the optical path to focus light onto a face of the encoder; and asecond optical system located between the encoder and the dispersiveelement to cause each of the plurality of light components of differentwavelengths to be focused onto an image pickup face of the first imagepickup device.
 5. The image pickup apparatus according to claim 2,wherein the encoder located between the object and the beam splitter,and the image pickup apparatus further includes an optical systemlocated between the encoder and the beam splitter to cause each of theplurality of light components of different wavelengths to be focusedonto a first image pickup face of the first image pickup device, andcause the other of the plurality of split light to be focused onto asecond image pickup face of the second image pickup device.
 6. The imagepickup apparatus according to claim 1, wherein the dispersive element islocated to cause a part of a light beam toward the at least one imagepickup device to be spatially dispersed into the plurality of lightcomponents of different wavelengths, the image pickup device includes aplurality of first light detection cells to receive the plurality oflight components of different wavelengths to acquire the first image, aplurality of second light detection cells to receive another part of thelight beam without passing through the dispersive element to acquire thesecond image, and an optical array located to face an image pickup faceof the image pickup device, the optical array causing the plurality oflight components of different wavelengths to enter the plurality offirst light detection cells and causing the other part of the light beamwithout passage through the dispersive element to enter the plurality ofsecond light detection cells.
 7. The image pickup apparatus according toclaim 1, further comprising: a signal processing circuit which generatesa plurality of images for respective wavelength bands of the light,based on the first image, the second image, and a spatial distributionof light transmittance in the encoder.
 8. The image pickup apparatusaccording to claim 7, wherein the signal processing circuit generatesthe plurality of images for the respective wavelength bands by astatistical method.
 9. The image pickup apparatus according to claim 7,wherein the number of pieces of data in the plurality of images for therespective wavelength bands of the light is larger than a sum of thenumber of pieces of data in the first image and the number of pieces ofdata in the second image.
 10. The image pickup apparatus according toclaim 7, wherein the signal processing circuit generates the pluralityof images for the respective wavelength bands by calculating a vector f′in accordance with the following formula:$f^{\prime} = {{\underset{f}{argmin}{{g - {Hf}}}_{l_{2}}} + {{\tau\Phi}(f)}}$where g is a vector whose elements contain signal values of a pluralityof pixels in the first image and signal values of a plurality of pixelsin the second image, H is a matrix which is determined by the spatialdistribution of light transmittance in the encoder and a spectralcharacteristic of the dispersive element, τΦ(f) is a regularizationterm, and τ is a weighting factor.
 11. A spectroscopic systemcomprising: an image pickup apparatus which includes an encoder locatedon an optical path of light from an object, the encoder including aplurality of first regions with first light transmittance and aplurality of second regions with second light transmittance lower thanthe first light transmittance, a dispersive element located farther awayfrom the object than the encoder on the optical path to cause light tobe spatially dispersed into a plurality of light components of differentwavelengths, and at least one image pickup device located to receive theplurality of light components of different wavelengths to acquire afirst image, in which the plurality of light components of differentwavelengths are superimposed in mutually shifted positions; and a signalprocessing apparatus which generates a plurality of images forrespective wavelength bands of light, based on the first image, thesecond image, and a spatial distribution of light transmittance in theencoder.
 12. A spectroscopic method comprising: spatially encodingintensity of light from an object; spatially dispersing at least part ofthe encoded light into a plurality of light components of differentwavelengths; acquiring a first image, in which the plurality of lightcomponents of different wavelengths are superimposed in mutually shiftedpositions; acquiring a second image, based on light from the objectwithout spatially dispersed in accordance with wavelength; andgenerating a plurality of images for respective wavelength bands, basedon the first image, the second image, and a pattern for the encoding.